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Abstract  -  This  paper  proposes  a  tele-instruction  system  for 
ultrasound  probe  operation,  based  on  shared  Augmented 
Reality  (AR)  technology.  Telemedicine,  a  strategy  to  diminish 
geographic  gaps  in  the  quality  of  medical  service,  is  fashionable. 
However,  existing  medical-data  transmission  systems  cannot 
convey  the  skills  of  physicians,  so  they  cannot  eliminate  the 
problem  of  geographic  gaps.  This  research  developed  a 
telemedicine  system  that  transports  physicians’  skills  over  a 
shared  AR  space.  The  shared  AR  technology  provides  an 
environment  where  users  can  exchange  spatial  information,  and 
facilitates  smooth  communication.  This  research  concentrates 
on  interfaces  for  physicians  and  technicians  at  the  patient  site. 
The  proposed  system  provides  intuitive  and  unrestricted 
interfaces  for  the  physician  and  technician,  using  a  data 
projector  and  an  LCD  tablet.  At  the  patient  site,  instructions  for 
the  technician  are  projected  directly  onto  the  patient’s  body, 
using  a  data  projector.  The  physician  site  allows  the  physician 
to  send  instructions  on  probe  operation  using  an  LCD  tablet. 
This  information  is  displayed  as  a  “web-mark”,  which  provides 
spatial  information  for  ultrasound  probe  operation. 
Furthermore,  the  system  sends  environmental  information 
about  the  patient  to  the  physician,  such  as  the  technician’s 
behavior  and  patient’s  posture,  via  an  immersive  display.  The 
proposed  system  was  tested  with  a  physician  using  real  tele¬ 
diagnosis.  The  results  demonstrated  the  effectiveness  of  the 
proposed  method  in  providing  smooth  communication  in 
telemedicine. 
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I.  Introduction 

Telemedicine  is  a  new  medical  service  model  that  enables  a 
patient  to  receive  medical  services  without  visiting  a  hospital 
by  connecting  medical  sites  and  patients.  Existing 
telemedicine  systems  use  conventional  multi-media 
communications  that  can  handle  medical  data,  such  as  vital 
signs,  X-ray  and  CT  images,  and  so  on  [1-2].  Such  systems 
require  physicians  or  medical  technicians  with  sufficient  skill 
to  provide  appropriate  medical  service  to  the  patient. 
However,  skilled  persons  are  not  always  available,  and  an 
unskilled  person  may  require  the  help  of  a  specialist  to  obtain 
appropriate  medical  data.  In  such  cases,  the  specialist  must 
transmit  instructions  on  how  to  handle  medical  devices  with 
the  utmost  care  and  patience,  as  direct  communication  using 
manual  manipulation  is  not  available  with  any  existing 
telemedicine  system.  This  paper  proposes  an  innovative 
telemedicine  system  that  enables  direct  instruction,  in  which 
the  participants  share  the  same  time  and  space  via  a  network. 


II.  Methods 

Telemedicine  systems  that  use  popular,  universal  medical 
devices  are  the  most  effective.  Therefore,  the  authors  are 
developing  a  telemedicine  system  that  uses  an  ultrasound 
diagnostic  device  that  is  widely  available,  even  in  small 
clinics  [3-4].  As  mentioned  above,  good  communication  is 
necessary  for  smooth  diagnosis.  With  the  ultrasound 
diagnostic  device,  the  most  important  information  required  to 
obtain  appropriate  medical  data  is  the  location  and  orientation 
of  the  probe  [5].  A  shared  Virtual  Reality  (VR)  space 
facilitates  smooth  communication,  because  it  allows  the 
exchange  of  spatial  information.  However,  a  shared  VR 
environment  requires  many  computational  and  network 
resources  to  virtualize  the  whole  environment.  Moreover, 
faults  in  the  VR  equipment  may  cause  a  fatal  error  in  medical 
treatment.  Augmented  Reality  (AR)  technology  can 
overcome  these  problems  [6];  the  participants  can  see  the  real 
patient  directly.  This  paper  describes  a  new  method  that 
focuses  on  the  transfer  of  information  concerning  probe 
operation  via  shared  AR  space. 

A.  System  Overview 

In  this  system,  the  physician  and  patient  sites  are  connected 
via  a  network.  The  system  exchanges  several  kinds  of  data, 
as  shown  in  Fig.  1. 

At  the  physician  site,  the  proposed  system  allows  the 
physician  to  ‘touch’  the  patient’s  body  using  an  LCD  tablet 

(Fig.  2). 
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Fig.  1 .  Data  exchange  flow  in  the  proposed  system 
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Fig.  2.  Interface  at  the  physician  site 

The  pen  substitutes  for  the  probe;  the  tablet  gives  the 
position,  and  the  magnetic  positioning  sensor  attached  to  the 
pen  determines  the  orientation.  At  the  patient  site,  a  data 
projector  projects  the  information  directly  onto  the  patient’s 
body,  as  shown  in  Fig.  3. 

This  interface  allows  users  to  show  and  see  information 
about  probe  operation  directly  on  the  patient’s  body,  enabling 
smooth  communication.  The  spatial  information  about  probe 
operation  is  5D,  consisting  of  2D  position  information  and 
3D  orientation  information.  In  order  to  show  this  5D 
information  on  a  2D  screen,  the  authors  designed  a  “web- 
mark”  as  a  pointer  to  relay  instructions.  Fig.  4  shows  the 
cobweb-shaped  pointer  used  as  the  web-mark. 

The  web-mark  shows  the  following  information: 

1)  The  center  is  where  to  place  the  probe. 

2)  Special  arc  1  indicates  a  “rotation  angle”  that  surrounds 
the  probe. 

3)  Special  arc  2  indicates  the  “slant  direction”  and  “slant 
angle”  of  the  probe. 


Fig.  5.  The  magnetic  sensor  pen 


B.  System  support  information 


The  physician  can  show  the  5D  spatial  information  by 
changing  these  three  features  of  the  web-mark  using  a 
specially  developed  pen  containing  a  magnetic  sensor,  as 
shown  in  Fig.  5. 


Fig.  3.  Interface  at  the  patient  site 


In  another  experiment,  we  identified  the  following  problems. 
When  the  web-mark  was  projected  on  the  side  of  the  body,  it 
was  distorted  and  “slant”  and  “rotate”  information  were  lost 
(Fig.  6).  To  solve  this  problem,  the  technician  sometimes 
needed  to  ask  for  instructions  repeatedly.  We  therefore  added 
support  information  to  the  web-mark  (see  Fig.  7). 


Fig.  6.  Lost  information 


Fig.  7.  Support  information 


The  support  information  provides  updated  instructions,  and 
was  added  to  the  web-mark,  as  shown  in  Fig.  8. 

Furthermore,  the  system  provides  information  about  the 
patient’s  environment  to  the  physician,  such  as  the 
technician’s  behavior  and  the  patient’s  posture,  using  an 
immersive  display  (Fig.  9).  The  immersive  display  consists 
of  three  94-inch  screens  covering  120  degrees,  which  is  the 
human  field-of-view. 


III.  Experiments 

The  prototype  system  was  evaluated  using  real-time  tele¬ 
diagnosis  of  heart  disease,  via  128-Kbps  ISDN  connections 
between  Nara  and  Sapporo,  and  Nara  and  Kurashiki,  and  at 
two  virtual  sites  in  our  laboratory  in  Nara  via  a  local  network. 
These  experiments  involved  physicians  (Fig.  10)  and 
postgraduate  students  of  the  Nara  Institute  of  Science  and 
Technology  who  filled  the  technician’s  role  (Fig.  11). 

In  this  test,  the  authors  evaluated  the  utility  of  the  proposed 
system,  and  the  accuracy  of  the  transmitted  instructions. 

IV.  Results 

Using  the  proposed  system,  the  physicians  were  able  to 
instruct  the  technicians  how  to  handle  the  ultrasound 
diagnostic  device  without  special  lessons. 

Using  video  cameras,  we  recorded  the  instruction  sequences 
between  physicians  and  technicians.  Then,  we  analyzed  the 
“comprehensibility”  and  “rapidity”  of  instructions. 


Fig.  10.  The  physician  site 


Fig.  9.  Immersive  display  at  the  physician  site 


Fig.  1 1 .  The  patient  site 


The  results  were  as  follows: 

A.  Comprehensibility  of  instructions 

1)  Proposed  system  (without  support  information) 

A)  Acquisition  “ Long  axis  view ”  of  the  chest 

All  instructions  (“Slide”,  “Rotate”  and  “Slant”)  were 
accurately  transmitted  to  the  patient  site. 

B)  Acquisition  “ Four  chamber  view ”  from  the  left  side 
The  students  sometimes  couldn’t  understand  the 
instructions,  which  then  had  to  be  repeated. 
Especially,  “Rotate”  and  “Slant”  seemed  difficult  to 
communicate. 

2)  Proposed  System  (with  support  information) 

A)  Acquisition  “ Long  axis  view ”  of  the  chest 

All  instructions  (“Slide”,  “Rotate”  and  “Slant”)  were 
accurately  transmitted  to  the  patient  site. 

B)  Acquisition  “ Four  chamber  view ”  from  the  left  side 
All  instructions  (“Slide”,  “Rotate”  and  “Slant”)  were 
accurately  transmitted  to  the  patient  site. 

Smooth  communication  was  realized  using  the  proposed 
telemedicine  system.  Without  support  information,  there 
were  some  difficulties  in  image  acquisition  from  the  side. 
However,  the  support  information  solved  this  problem. 

B.  Rapidity  of  instruction 

The  results  (Fig.  12)  depended  on  which  of  three  methods 
of  instruction  were  used:  Instruction  by  voice  only, 
instruction  by  voice  and  the  proposed  system  without  the 
immersive  display,  and  instruction  by  voice  and  the  proposed 
system  with  the  immersive  display. 


Type  of  Instruction 
Fig.  12.  Instruction  time 

Using  the  proposed  system,  the  required  time  decreased 
drastically.  Although  no  definite  difference  was  found  with 
the  immersive  display,  the  physicians  commented  that  the 
instructions  were  most  easily  explained  using  the  immersive 
display. 


V.  Discussion 

In  this  paper,  we  propose  a  new  telemedicine  system  using 
shared  AR  space  between  diagnosis  and  measurement  sites. 
Using  the  proposed  system,  a  student  with  no  experience  was 
able  to  use  an  ultrasound  probe  to  acquire  appropriate 
ultrasound  images. 

The  proposed  telemedicine  system  realized  smooth 
communication  and  an  intuitive  interface.  However,  some 
problems  occurred  in  the  first  trial  when  acquiring  images 
from  the  side,  because  the  web-mark  was  distorted  due  to  the 
non-flat  display  surface,  i.e.,  the  patient’s  body.  The 
experimental  results  confirmed  that  support  information 
effectively  resolved  this  problem.  Although  a  definite 
difference  wasn’t  found,  the  immersive  display  seemed  to  be 
more  useful  for  tele-instruction. 

The  authors  believe  that  the  proposed  system  will  increase 
the  quality  of  medical  services  in  the  near  future. 
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